swarm/api: support mounting manifests via FUSE #3690

jmozah · 2017-02-19T16:28:10Z

This PR adds mounting swarm hash/ensname as a fuse fie system.
It adds new commands to swarm cmdline

swarmfs.mount (manifest-hash/ensname, mountpoint)
swarmfs.listmounts ()
swarmfs.unmount (mountpoint)

Tested in linux and osx.

Requires fuse to be installed for this to work.
Mounts the FS as read only for now

mention-bot · 2017-02-19T16:28:11Z

@jmozah, thanks for your PR! By analyzing the history of the files in this pull request, we identified @zsfelfoldi, @fjl and @karalabe to be potential reviewers.

jmozah · 2017-02-19T16:39:45Z

This also supports random seek on a big file making full use of the lazyreader in swarm.
Need to add "force unmount/ external unmount" while quitting swarm...

zelig · 2017-02-23T12:45:39Z

reviewing this now

GitCop · 2017-02-24T07:46:17Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: c100b2f8b3aa6cb7ef4814edab286d525b0d7f97
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-02-24T08:21:51Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: c100b2f8b3aa6cb7ef4814edab286d525b0d7f97
Commits must be prefixed with the package(s) they modify
Commit: 46fd61e3619cf13848503ed124479bc2b856d4b5
Commits must be prefixed with the package(s) they modify
Commit: 82d421d7fdf4b5cc9c0e574c7b31339b03cc8ae6
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-02-24T08:45:34Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: c100b2f8b3aa6cb7ef4814edab286d525b0d7f97
Commits must be prefixed with the package(s) they modify
Commit: 46fd61e3619cf13848503ed124479bc2b856d4b5
Commits must be prefixed with the package(s) they modify
Commit: 82d421d7fdf4b5cc9c0e574c7b31339b03cc8ae6
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

jmozah · 2017-02-24T09:22:06Z

Pulled recent changes
fixed this PR w.r.to new changes (glog to log)
added fuse in vendor

jmozah · 2017-02-24T16:31:28Z

@zelig Need some fixes based on OS (compilation is failing in windows as Fuse is not supported in windows). Will fix that and rebase it.

GitCop · 2017-02-24T17:25:44Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 11c07d58723a2fa5cdf138959308607ac1334a7d
Your commit message body contains a line that is longer than 80 characters

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-02-24T17:30:55Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 11c07d58723a2fa5cdf138959308607ac1334a7d
Your commit message body contains a line that is longer than 80 characters

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-02-24T17:38:33Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 7f2cea16205b6a8e152a862f888e52e67e2d4b05
Your commit message body contains a line that is longer than 80 characters
Commit: 3d30d8eafa298a85efd5d9709cb275c31d904aca
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-02-24T17:43:59Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 7f2cea16205b6a8e152a862f888e52e67e2d4b05
Your commit message body contains a line that is longer than 80 characters

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

karalabe · 2017-02-27T17:10:55Z

internal/web3ext/web3ext.go

 }

+const SWARMFS_JS = `


Please keep the list is this file sorted alphabetically. Both the above and below too (swarm is notorious for adding stuff all over the place in this file :P)

sure.. will do that

GitCop · 2017-03-01T13:50:29Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 3fc7e66fceec349e8c6f1ce978404387cdf02b28
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-03-01T16:11:37Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: a9b045ed6d00d714d93901c795a9f6b5e09d827a
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-03-01T17:20:57Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: cf8da3ac1fdca196c362b0941dc83f49dd9c70b0
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-03-02T14:38:42Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 5c0220ab969792f5710540dc98acb32b2f7fbd79
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-03-02T15:51:09Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: f5a25911db3bf885af8794eb2bed0b1c72fb7c1b
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-03-03T06:22:25Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 8679b232a617ce5eee547b7be566d40863de7b59
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-03-03T09:02:29Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 8679b232a617ce5eee547b7be566d40863de7b59
Commits must be prefixed with the package(s) they modify
Commit: c78c6ff932124d62aa129e85ddd1b47dea072911
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-03-03T10:48:57Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 8679b232a617ce5eee547b7be566d40863de7b59
Commits must be prefixed with the package(s) they modify
Commit: c78c6ff932124d62aa129e85ddd1b47dea072911
Commits must be prefixed with the package(s) they modify
Commit: b2be8237ced4cfc8444fa03ac9eebbc12cb78911
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-03-03T12:57:41Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 8679b232a617ce5eee547b7be566d40863de7b59
Commits must be prefixed with the package(s) they modify
Commit: c78c6ff932124d62aa129e85ddd1b47dea072911
Commits must be prefixed with the package(s) they modify
Commit: b2be8237ced4cfc8444fa03ac9eebbc12cb78911
Commits must be prefixed with the package(s) they modify
Commit: 4b9ab71114aa3755df76fbc20d9cd6d9a673451f
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-03-03T13:55:55Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: b2fbd99e5ee0d8ff5d8ae2d64cd4c1f4f2ff69ad
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-03-03T14:47:30Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 94fee69d96c90a67659d75fde3c68e88bc8f722c
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-03-03T15:17:46Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 449151d8c038d6d8dc2ef30edd723a5bae9a6a74
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

GitCop · 2017-03-04T08:11:27Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 797a176ae633de9a72b2cad1930a72db1e0178e3
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

zelig

just very minor things.
Otherwise great stuff, thanks @jmozah

I need someone else to approve the travis and vendor stuff. I guess @karalabe might be willing

zelig · 2017-03-06T12:49:22Z

swarm/api/swarmfs_unix.go

+
+
+
+func (self *SwarmFS) Mount(mhash, mountpoint string) bool {


RPC supports (interface{}, err) composite return values. I think Mount and Unmount should use it, for proper RPC error responses

yeah.. will fix this

zelig · 2017-03-06T12:54:29Z

swarm/swarm.go

@@ -216,7 +220,7 @@ func (self *Swarm) Stop() error {
 	if self.lstore != nil {
 		self.lstore.DbStore.Close()
 	}
-
+	self.sfs.Stop()


can something foul happen if we do not unmount active mounts. E.g. KILLSIG?
SIGINT and recently also SIGTERM are caught gracefully

If the device is busy.. we cannot unmount and this will return after 5 seconds without unmounting. We cannot do much here... the mount will be visible from "mount" command and it can be unmounted using external "umount" command ....

we can always unmount from the bash prompt "umount swarmfs"

zelig · 2017-03-06T12:56:19Z

swarm/swarm.go

+			Namespace: "swarmfs",
+			Version:   api.Swarmfs_Version,
+			Service:   api.NewSwarmFS(self.api),
+			Public:    true,


well I think the idea was that we do not 'public' an API that can read/write the filesystem.
Not sure if there is rpcapis

aah.. ok.. i will change it to false...

zelig · 2017-03-06T13:01:21Z

also the travis issue with sudo? is that gonna be fixed? @karalabe
https://travis-ci.org/ethereum/go-ethereum/jobs/207629031#L295

fjl

Please fix the (minor) issues. I still need to actually try it out but the code looks OK.

General feedback: The fuse stuff should probably be in its own package e.g. swarm/fusefs.

fjl · 2017-03-16T02:51:34Z

swarm/api/swarmfs.go

+	})
+	return swarmfs
+
+}


On any but the first call, NewSwarmFS returns the same SwarmFS regardless of api. IMHO that's confusing and might lead to bugs later. Why does it need to be a global singleton?

i thought about this... let me see if there is clean way to avoid having singleton logic.

I tried moving the fuse stuff inside a package.. but this right now depends on some methods which are not exposed as api... let me attempt this once more ..

fjl · 2017-03-16T02:51:49Z

swarm/api/swarmfs_unix.go

@@ -0,0 +1,266 @@
+// +build !windows


Please put build constraints below the license header.

// Copyright... // +build !windows

aah okay.. will fix it

fjl · 2017-03-16T02:56:39Z

build/ci.go

+				packages = append(packages, strings.TrimSpace(line))
+			}
+		}
+	}


IMHO it would be better to remove anything with github.com/ethereum/go-ethereum/vendor/ prefix, on all platforms.

It cannot be removed for all platforms, as some fuse test are dependent on it.
It is removed for windows since windows does not support fuse or user level file system support in general.

We should not remove for all platforms since "vendor/basil.org/fuse" is required by other platforms for the test cases to work.
Fuse support is not there only in windows.. thats why i removed those for windows.

If I understand correctly, this change fixes make all and related commands by excluding "bazil.org/fuse" from the expansion of "./..." for the purpose of building and testing.

My suggestion is that all vendored packages should be excluded. Building vendored packages explicitly is almost never needed; testing them doesn't do anything because we strip their _test.go files when adding them. They will still be built as dependencies of non-vendored packages.

I gotta say that it sucks that bazil.org/fuse doesn't even build on Windows.

aah... i didnt see this recent comment...
Are you saying that i remove compiling vendor packages for all platforms...?

fixed it.. i removed all vendors packages

zelig · 2017-03-16T03:29:33Z

@jmozah please consider @fjl's advice.

I am very happy with this PR

@fjl I tried it out and it works like a charm

jmozah · 2017-03-22T15:40:58Z

@fjl @zelig Fixed the following comments

Moved the singleton logic out
Put build constraints below the license header

Thanks for reviewing and your comments

fjl · 2017-03-22T16:10:17Z

Minor notes about the commit message (you don't need to fix this, I'll edit it when squashing).

You don't need to include every package that was modified. It is sufficient to include the packages where the most of the new stuff is. In this case, that's probably "cmd/swarm, swarm/api".
The format is "pkg1, pkg2: foo", not "pkg1;pkg2 : foo".

fjl · 2017-03-22T16:10:52Z

And a question: are you planning to move fuse support to swarm/fusefs or not?

jmozah · 2017-03-22T16:47:56Z

@fjl Not in this release... That will require some refactoring of the code in swarm/api and expose some methods which are not exposed for now.

jmozah · 2017-03-22T16:51:01Z

@fjl I am doing more refactoring and adding more stuff in #3751.. will take it up as part of that if that is okay

fjl · 2017-03-22T16:53:22Z

Cool, thanks. I'm fine with keeping it in swarm/api for now.
.travis.yml has been updated on master. Sorry for the rebase trouble.
Please implement the vendor/ skipping as suggested. We're good to merge after that.

jmozah · 2017-03-22T16:59:01Z

@fjl no issues.. i will resolve the conflict and rebase
Regarding vendor skipping my answer was that we cannot skip it for all platforms...
They are used in testcases for linux and darwin...
thats the reason i skipped only for windows...

jmozah · 2017-03-22T17:11:03Z

@fjl fixed the vendor skipping..

zelig changed the title ~~rpc;swarm;cmd/swarm;swarm/api : Fuse support for swarm~~ FUSE read-only support for swarm Feb 23, 2017

zelig assigned jmozah Feb 23, 2017

zelig added feature labels Feb 23, 2017

jmozah closed this Feb 24, 2017

jmozah reopened this Feb 24, 2017

karalabe reviewed Feb 27, 2017

View reviewed changes

zelig suggested changes Mar 6, 2017

View reviewed changes

zelig added this to the 1.6.0 milestone Mar 13, 2017

fjl requested changes Mar 16, 2017

View reviewed changes

swarm/api : Fuse read-only support

be1ed5d

fjl changed the title ~~FUSE read-only support for swarm~~ swarm/api: support mounting manifests via FUSE Mar 23, 2017

fjl approved these changes Mar 23, 2017

View reviewed changes

fjl merged commit 11e7a71 into ethereum:master Mar 23, 2017

This was referenced Mar 23, 2017

build: unify vendor skipping, always run go vet #3813

Merged

swarm: mount a swarm manifest as a local filesystem using fuse #3539

Closed

jmozah deleted the fusefs branch March 25, 2017 09:00

adamschmideg added type:feature and removed type:feature in progress labels Dec 14, 2018

		}

		const SWARMFS_JS = `

swarm/api: support mounting manifests via FUSE #3690

swarm/api: support mounting manifests via FUSE #3690

Conversation

jmozah commented Feb 19, 2017 • edited Loading

mention-bot commented Feb 19, 2017

jmozah commented Feb 19, 2017

zelig commented Feb 23, 2017

GitCop commented Feb 24, 2017

GitCop commented Feb 24, 2017

GitCop commented Feb 24, 2017

jmozah commented Feb 24, 2017

jmozah commented Feb 24, 2017

GitCop commented Feb 24, 2017

GitCop commented Feb 24, 2017

GitCop commented Feb 24, 2017

GitCop commented Feb 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GitCop commented Mar 1, 2017

GitCop commented Mar 1, 2017

GitCop commented Mar 1, 2017

GitCop commented Mar 2, 2017

GitCop commented Mar 2, 2017

GitCop commented Mar 3, 2017

GitCop commented Mar 3, 2017

GitCop commented Mar 3, 2017

GitCop commented Mar 3, 2017

GitCop commented Mar 3, 2017

GitCop commented Mar 3, 2017

GitCop commented Mar 3, 2017

GitCop commented Mar 4, 2017

zelig left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zelig commented Mar 6, 2017

fjl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fjl Mar 22, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zelig commented Mar 16, 2017

jmozah commented Mar 22, 2017

fjl commented Mar 22, 2017 • edited Loading

fjl commented Mar 22, 2017

jmozah commented Mar 22, 2017 • edited Loading

jmozah commented Mar 22, 2017 • edited Loading

fjl commented Mar 22, 2017

jmozah commented Mar 22, 2017

jmozah commented Mar 22, 2017

jmozah commented Feb 19, 2017 •

edited

Loading

fjl Mar 22, 2017 •

edited

Loading

fjl commented Mar 22, 2017 •

edited

Loading

jmozah commented Mar 22, 2017 •

edited

Loading

jmozah commented Mar 22, 2017 •

edited

Loading